ca 

Conseils  Facteurs  Humains 

Human  Factors  Consulting 


THALES 


12om  Methodology: 
Process  vl.1 

Prepared  by: 

Thales  Canada,  Defence  and  Security 

1405  boul.  du  Parc-Technologique 
Quebec,  QCG IP  4P5 


C3  Human  Factors  Consulting  Inc. 

1405,  boul.  Du  Parc-Technologique 
Quebec,  QC,  G1P  4P5 


DRDC-RDDC-2014-C193 
31  March  2014 

The  scientific  or  technical  validity  of  this  Contract  Report  is  entirely  the  responsibility  of  the 
Contractor  and  the  contents  do  not  necessarily  have  the  approval  or  endorsement  of  the 

Department  of  National  Defence  of  Canada. 

Thales  Document  Control  Number  (DCN):  2014C.005-REP-03-AT5  Rev.  01 
PWGSC:  W7701-10307/001/QCL 


Authors:  Daniel  Lafond  Scientific  Authority:  Natalia  Derbentseva 

Jean-Frangois  Gagnon  (416)  635-2000x2302 

Marie-Eve  St-Louis 
Sebastien  Tremblay 


12om  Process 


ci 


£ 


THALES 


ABSTRACT 

The  12om  project  seeks  to  develop  a  methodology  (i.e.,  process  and  tools]  to  improve  the 
understanding  of  a  complex  situation  by  a  multidisciplinary,  government-wide  team.  The 
specific  objective  of  this  report  is  to  describe  and  document  the  12om  process  (vl.l].  The 
12om  process  aims  to  support  teamwork  and  performance  in  the  context  of  the  current 
Canadian  Forces  Operational  Planning  Process  (CFOPP),  particularly  during  joint  civil- 
military  planning  endeavours,  such  as  whole-of-government  approach  initiatives.  The 
present  document  is  divided  into  three  sections.  First,  it  introduces  the  context  of  the 
project  and  outlines  the  method  employed  for  the  development  of  the  12om  process. 
Second,  informed  by  multiple  sources,  this  report  describes  version  1.1  of  the  12om 
process.  This  process  includes  three  main  dimensions:  1)  A  toolbox  approach;  2]  Training; 
and  3)  Inclusion  of  an  inter-agency  planning  specialist/facilitator.  There  is  also  a  discussion 
of  the  sub-processes  underlying  the  12om  methodology  components.  These  components 
include  the  Whole-of-Government  OPP  handbook,  the  team  building  and  handover 
procedure,  the  interactive  common  glossary,  IMAGE  v3  [i.e.,  individual  and  collaborative 
diagrams),  the  cross-impact  method,  the  Op  Design  tool,  and  the  mission  analysis  briefing 
template.  Finally,  in  section  three,  the  main  features  of  Process  vl.l,  its  limitations,  and 
possible  next  steps  are  discussed. 
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EXECUTIVE  SUMMARY 


This  work  was  conducted  in  support  of  the  Applied  Research  Project  (ARP)  12om  entitled 
"Collaborative  Understanding  of  Complex  Situations".  The  overall  purpose  of  this  project  is 
to  develop  a  methodology  (i.e.,  a  combination  of  processes  and  support  tools)  to  improve 
the  understanding  of  a  complex  situation  by  a  multidisciplinary  team  combining  experts 
from  different  governmental  departments.  This  document  presents  the  results  of  work  to 
develop  the  12om  process  in  three  iterations,  with  the  specific  aim  to  support  teamwork 
and  performance  in  the  context  of  the  current  Canadian  Forces  Operational  Planning 
Process  (CFOPP),  in  situations  of  interagency  planning  such  as  Whole-of-Government  or 
Comprehensive  Approach  initiatives. 

The  specific  objective  of  the  current  report  is  to  describe  version  1.1  of  the  12om  process  - 
the  third  and  final  iteration  planned  within  the  scope  of  Project  12om.  This  iteration  was 
performed  following  the  completion  of  the  Limited  Objective  Experiment  (LOE)  #2  and  was 
informed  by  the  pattern  of  results  collected  during  the  LOE  (see  ATT7  report  "12om  LOE 
#2:  Final  Results  Summary  and  Recommendations”). 

Process  vl.l  builds  upon  the  toolbox  metaphor  from  Process  v0.2  (reported  elsewhere,  see 
ATT5  report  "Development  of  the  IMAGE  process:  Version  0.2).  It  complements  the  toolbox 
by  providing  details  about  the  strength  and  weaknesses  of  each  tool,  informing 
commanders  and  interagency  planning  team  members  on  their  relative  utility  given  the 
context  of  their  mission.  The  process  specifically  highlights  components’  use  and  training 
time/effort  as  constraints,  but  also  underlines  their  potential  benefits  to  various 
collaborative  understanding  dimensions.  Moreover,  Process  vl.l  appreciates  individual 
components  both  in  terms  of  flexibility  and  impact  on  taskwork  to  perform.  Altogether, 
these  assessments  and  recommendations  about  the  toolbox  are  believed  to  provide  great 
insights  to  the  CF  and  other  Canadian  agencies  and  departments  on  the  use  of  the  12om 
methodology. 
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1  Introduction 

The  document  presents  the  12om  process,  resulting  from  three  major  development 
iterations  (version  0.1,  0.2,  and  1.1).  The  12om  process  aims  to  support  interagency 
teamwork  and  performance  in  general  and  in  the  context  of  the  current  Canadian  Forces 
Operational  Planning  Process  (CFOPP)  specifically.  This  work  was  conducted  in  support  of 
the  Applied  Research  Project  (ARP)  12om  entitled  "Collaborative  Understanding  of 
Complex  Situations”.  The  overall  purpose  of  this  project  is  to  develop  a  methodology  (i.e.,  a 
combination  of  processes  and  support  tools)  to  improve  the  understanding  of  a  complex 
situation  by  a  multidisciplinary  team  combining  experts  from  different  governmental 
departments. 

1.1  Background 

Collaborative  mission  analysis  is  an  inherently  difficult  enterprise  especially  in  a  whole  of 
government  (WoG)  planning  context  (e.g.,  Essens  et  al.,  2013;  Patrick  &  Brown,  2007).  The 
12om  project  seeks  to  identify  and  address  these  challenges  through  the  following 
technical  objectives: 

1.  Identify  support  requirements  in  Phase  1  of  the  case  study,  which  will  provide  input 
into  the  design  and  development  of  support  tools  and  processes  to  improve  team 
collaboration  within  this  type  of  context;  and 

2.  To  pilot  the  developed  methodology  in  Phase  2  of  the  case  study. 

The  project  aims  to  improve  the  understanding  of  a  complex  situation  by  a 
multidisciplinary  team  and  the  specific  collaboration  context  selected  is  a  J5  integrated  (or 
WoG)  planning  group.  Furthermore,  to  keep  the  scope  of  the  project  manageable,  the 
project  focus  is  on  the  mission  analysis  (orientation  phase  of  the  operational  planning 
process)  and  course  of  action  (COA)  development  phases  of  the  CFOPP.  These  two  stages 
were  chosen  as  a  focus  because  most  inter-agency  collaboration  occurs  during  these  two 
stages  ofthe  CFOPP  (Turnbull  &  Ulrich,  2013). 

1.2  Operational  Planning  Process 

The  CFOPP  is  comprised  of  five  main  stages  (see  Figure  1): 

•  The  Initiation  stage  results  in  the  activation  of  the  planning  staff  and  the 
commander’s  guidelines  about  the  kind  of  planning  process  to  achieve; 

•  The  Orientation  stage  results  in  the  development  of  the  commander's  planning 
guidance.  At  this  stage,  the  commander  orients  his/her  staff  towards  the 
determination  of  the  nature  of  the  problem  and  the  confirmation  of  the  results  to 
be  achieved; 

•  The  COA  Development  stage  results  in  the  production  of  the  CONOPS  (CONcept  of 
OPerationS)  that  identifies  the  commander’s  line  of  action  in  order  to  accomplish 
his/her  mission.  It  presents  the  COA  that  will  be  implemented; 
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•  The  Plan  Development  stage  results  in  a  set  of  orders  based  on  the  commander’s 
decision  to  provide  subordinate  and  supporting  units  with  all  of  the  necessary 
information  to  initiate  the  planning  or  the  execution  of  operations;  and 

•  The  Plan  Review  stage  results  in  a  regular  review  of  the  plan  to  evaluate  its 
viability.  The  review  period  of  the  plan  depends  on  the  evolution  of  the  situation, 
the  type  of  operation  and  the  environment 


- \ 

Initiation 

v _ 


( - A 

Orientation 

v _ / 


Course  of 
Action 

^  Development  ^ 


/ - \ 

Plan 

Development 
v _ y 


Figure  1:  Five  phases  of  the  Canadian  Forces  Operational  Planning  Process 

Figure  1  also  illustrates  an  ideal  state  in  which  the  whole  OPP  is  performed  flawlessly  in 
terms  of  the  quality  and  timeliness  all  five  components  of  the  process  (i.e.,  a  straight 
arrow).  However,  the  inherent  complexity  of  planning  within  a  WoG  context  makes 
accomplishing  this  ideal  state  extremely  unlikely.  Rather,  sub-optimal  team  dynamics,  poor 
shared  awareness  and  high  levels  of  workload  (amongst  other  team-centred  factors)  are 
likely  to  lead  to  sub-optimal  planning  performance  (e.g.,  Turnbull  &  Ulrich,  2013,  see  also 
Figure  2  for  an  example). 
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Figure  2:  Team-centred  factors  leading  to  sub-optimal  Orientation  and  COA  Development  performance 

The  overall  purpose  of  the  12om  project,  therefore,  is  to  develop  and  evaluate  a 
methodology  -  by  which  we  mean  a  coordinated  suite  of  process  improvements  and 
support  tools  -  to  improve  collaboration  in  multidisciplinary  teams.  The  context  in  which 
the  methodology  is  tested  is  the  CFOPP  within  a  J5  WoG  planning  group.  For  example, 
Figure  3  illustrates  a  hypothetical  scenario  within  which  a  combination  of  process 
improvements  and  support  tools  have  been  used  to  address  the  planning  deficiencies 
identified  in  Figure  2.  In  doing  so,  the  overall  CFOPP  process  performance  is  improved  to 
an  acceptable  level. 
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Process  Improvement  Process  Improvement  Tool  Support 


Tool  Support 


Figure  3:  Illustration  of  how  the  12om  project  seeks  to  improve  the  CFOPP  through  the  development  and  application 

of  a  novel  Methodology 

1.3  12om  Project  Objectives 

One  frequent  criticism  of  the  standard  CFOPP  process  is  that  it  is  not  very  well  suited  to 
deal  with  complex  situations  (Lauder,  2009).  Complexity  can  lead  to  poor  quality  decisions 
because  planners  might  possess  a  different  understanding  of  the  situation,  misunderstand 
their  commander’s  intent,  or  are  not  clear  about  the  roles,  relationships,  tasks,  and  success 
criteria.  Design  can  be  seen  as  a  critical  thinking  method  that  is  aligned  with  the  principles 
and  objectives  of  the  comprehensive  approach  to  military  decision  making.  The  so-called 
"art  of  design”  (Banach,  2009;  Banach  &  Ryan,  2009)  aims  at  helping  planners  to  better 
understand  the  operational  environment  (i.e.,  sensemaking ),  analyze  the  problem  space, 
and  consider  potential  solutions  so  they  can  exploit  opportunities  and  identify 
vulnerabilities.  The  methodology  developed  as  part  of  the  12om  project  should,  therefore, 
maximize  the  fit  across  the  processes  involved  in  collaborative  sense-making,  the 
techniques  for  knowledge  representation,  and  the  tools  required  to  support  the  expression 
and  sharing  of  this  knowledge  across  the  planning  team. 

In  summary,  the  12om  project  seeks  to  develop  a  methodology  (i.e.,  process  and  tools)  to 
improve  the  understanding  of  a  complex  situation  by  a  multidisciplinary,  government-wide 
team.  The  overall  objectives  of  project  are  to: 

1.  Develop  a  process  based  on  an  understanding  of  the  collaborative  process  through 
which  expert  teams  understand  complex  situations; 

2.  Investigate  and  develop  tools  to  support  this  process;  and 

3.  Empirically  evaluate  the  methodology  (i.e.,  a  coordinate  suite  of  process 
improvements  and  support  tools)  developed. 

The  specific  objective  of  the  work  reported  in  this  report  was  to  inform  the  development  of 
the  three  major  iterations  of  the  12om  methodology  (i.e.,  a  combination  of  tools  and 
processes)  based  on  support  requirements  captured  from  the  following  sources: 

1.  Empirical  evidence  from  the  LOE  #1  and  LOE  #2; 

2.  Subject  Matter  Expert  (SME)  feedback  obtained  during  and  after  LOE  #1; 

3.  Tool  and  process  recommendations  developed  as  part  of  the  previous  report 
entitled  "State-of-the-Art  on  Operational  Design  Processes  and  Representation 
Techniques”  (2014C.003-REP-01-AT3); 
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4.  Subjective  assessments  of  the  tool/process  feasibility  and  impact  on  the  OPP  by  the 
project  team;  and 

5.  Subjective  assessments  of  the  tool/process  feasibility  and  impact  on  the  OPP  by 
domain  experts.  The  data  was  collected  through  a  series  of  workshops. 

The  methodology  used  by  the  project  team  to  capture,  consolidate,  prioritise  and 
synthesize  the  tool  and  process  ‘solutions’  identified  during  the  earlier  phases  of  the 
project  is  described  in  detail  in  the  following  section. 

1.4  Process  Development  Methodology 

The  following  section  describes  the  general  methodology  that  was  used  to  develop  each  of 
the  three  12om  process  iterations  (vO.l,  v0.2  and  vl.l).  The  methodology  employed  for 
Process  v.01  is  presented  in  Figure  4  and  explained  in  detail  below. 


Figure  4:  Methodology  used  to  develop  Process  vO.l 


•  Consolidate  Issues :  The  first  step  was  to  consolidate  the  list  of  issues  identified  by 
both  the  empirical  results  from  the  LOE  #1  (and  Task  4.1  Analysis)  and  comments 
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from  the  SMEs  themselves  recorded  during  and  after  the  experimental  sessions. 
These  issues  were  then  grouped  into  similar  themes  and  the  source(s)  of  the  issues 
(e.g.,  experimental  metric,  SME  comment,  etc.)  recorded  and  tabulated. 

•  Identify  Solutions:  The  second  step  was  to  identify  and  consolidate  solutions  from 
both  the  ATT3  report  "State-of-the-Art  on  Operational  Design  Processes  and 
Representation  Techniques”  and  feedback  received  from  SMEs  during  de-briefing 
sessions  conducted  immediately  after  LOE  #1.  Once  again,  this  information  was 
recorded  in  table  format  to  expedite  the  prioritisation  process  of  step  3. 

•  Prioritise  Solutions:  The  solutions  identified  in  step  2  were  then  prioritised  against 
several  criteria,  including  Mandatory,  Primary,  Secondary  and  Tertiary  criteria  in 
terms  of  importance  for  the  current  project.  Each  solution  was  rated  against  the 
criteria  by  the  project  team  in  order  to  identify  a  short-list  of  solutions  for  further 
analysis. 

•  Conduct  Solution  Coherence  Analysis:  We  analysed  the  interrelationships  between 
the  short-listed  solutions  in  terms  of  identifying  both  general  themes  of  support,  as 
well  as  dependencies  and  incompatibilities  between  them.  The  intention  was  to  get 
a  sense  of  the  coherence  of  solutions  to  work  out  recommendations  arriving  at  a 
harmonious  set  of  tools  and  processes. 

•  Describe  Process /  Tools:  The  last  step  was  to  describe  the  short-listed  tools  and 
processes  in  more  detail  in  order  to  support  both  the  implementation  of  these 
solutions  for  LOE  #2,  and  the  development  of  discussion  points  for  the  forthcoming 
SME  workshop.  In  addition,  this  work  will  support  (and  be  informed  by)  parallel 
activities  pertaining  to  Task  6  "Identification  of  Tools”. 

•  Workshop  protocol:  Finally,  a  series  of  workshops  (task  4.3)  was  conducted  to  get 
feedback  from  SMEs.  The  comments  were  used  to  refine  the  initial  version  of  the 
12om  process  (vO.l)  into  its  second  iteration  (v0.2). 

The  methodology  employed  for  the  development  of  Process  v0.2  involved  creating  a  short¬ 
list  of  potential  components  and  presenting  it  in  a  series  of  workshops  to  SMEs  from  the  CF, 
CIDA,  DFAIT  and  Calian.  Two  workshop  cycles  were  conducted.  Cycle  one  involved  four 
distinct  events  (Workshop  la,  lb,  lc,  and  Id)  whereas  cycle  two  involved  a  single  event 
(Workshop  2)  after  revising  candidate  components  based  on  Cycle  1  results.  Workshop  2 
made  use  of  a  serious  game  scenario  (Peacemaker),  to  create  a  relevant  context  for  the 
hands  on  methodology  component  exercises  without  requiring  excessive  reading  time  from 
the  participants.  Each  workshop  event  was  conducted  in  a  small  group  with  two  to  eight 
SMEs.  Workshops  involved  domain  experts  from  DFAIT  (Workshop  la  and  2),  CF 
(Workshop  lb  and  2),  Calian  (Workshop  lc),  and  CIDA  (Workshop  Id  and  2).  Each  event 
consisted  of  several  activities  including  presentation  of  the  results  of  the  first  experiment 
(and/or  previous  workshops),  presentations  and  demonstrations  of  the  methodology 
components  by  the  research  team,  focus  group  discussions  identifying  target  areas  for 
process  and  tool  support,  hands-on  trials  of  the  methodology  components,  participants’ 
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evaluation  and  suggestions  for  improvement,  and  functional  gap  analysis.  Results  from 
these  workshops  provided  the  necessary  insights  to  achieve  the  development  Process  v0.2. 
A  dry-run  serving  to  pre-test  the  12om  Process  and  Tools  with  Calian  personnel  allowed 
fine-tuning  Process  v0.2  in  preparation  for  L0E#2. 

The  development  of  Process  vl.l  was  based  on  the  results  from  LOE  #2.  LOE  #2  involved,  a 
simulated  WoG  effort  carried  out  to  respond  to  a  polio  outbreak  in  the  horn  of  Africa  [i.e., 
Somaliland,  Ethiopia,  Djibouti).  A  WoG  team  of  five  members  was  created  for  that  purpose 
[three  from  the  CF  [J5,  J52;  J50ps]  and  two  civilians  [J5Dev  and  J5Gov]).  Four  SME 
observers  also  participated  in  the  LOE  #2,  two  of  them  playing  the  role  of  ROC  and 
Commander.  The  detailed  LOE  procedure,  measurements  and  results  are  described  in 
report  2014C.007-REP-04-AT7  [see  also  Appendix  A  for  key  observations  with  regards  to 
12om  Process  vl.l).  Following  LOE  #2,  the  development  of  Process  vl.l  involved 
extracting  key  results  [e.g.,  collaboration  pattern,  strengths,  weaknesses,  tool  synergies)  to 
guide  the  design  of  the  final  process  improvements.  Using  the  MYRIAD  multi-criteria 
analysis  tool,  we  designed  a  preference  model  that  combines  different  key  results  into  a 
coherent  assessment  of  the  overall  12om  methodology  and  its  components.  Furthermore, 
MYRIAD  allowed  performing  a  sensitivity  analysis  in  order  to  derive  the  most  promising 
areas  for  improvement. 

1.5  Document  Overview 

This  document  describes  the  work  performed  to  develop  the  12om  process.  The  document 
is  organized  as  follows: 

•  Section  One:  Introduction.  This  section  provides  the  background,  purpose  and 
objectives  of  the  study,  the  process  development  methodology,  together  with  an 
overview  of  the  document; 

•  Section  Two:  Process  vl.l.  This  section  presents  Process  vl.l  of  the  12om 
methodology.  This  process  includes  three  main  dimensions:  1)  A  toolbox 
approach;  2)  Training;  and  3)  Inclusion  of  an  inter-agency  planning 
specialist/facilitator. 

•  Section  Three:  Conclusion.  This  section  summarizes  the  main  features  of  the 
12om  process,  reviews  the  main  limitation  of  the  approach  as  it  is  now,  and 
discusses  the  next  steps  of  this  research  endeavour. 
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2  12om  Process  vl.1 

The  overall  purpose  of  this  project  is  to  develop  a  methodology  (i.e.,  a  combination  of 
process  and  support  tools)  to  improve  the  understanding  of  a  complex  situation  by  a 
multidisciplinary  team  combining  experts  from  different  governmental  departments.  More 
precisely,  the  methodology  aims  to  support  teamwork  and  performance  in  the  context  of 
the  current  CFOPP.  This  section  describes  the  process  of  the  12om  methodology. 

The  12om  process  is  based  on  the  toolbox  metaphor.  According  to  this  metaphor,  a  series 
of  "simple”  tools,  when  used  in  the  right  context,  outweighs  the  benefits  of  a  single 
"complex”  tool  aiming  to  be  generically  applied  to  various  contexts.  In  that  regard,  the 
components  integrated  within  the  12om  methodology  are  not  intrinsically  good  or  bad,  but 
must  be  used  in  an  adaptive  manner.  The  process  explicitly  states  specific  conditions  under 
which  each  component  should  provide  effective  support  When  applying  these  conditions 
in  the  context  of  OPP,  a  natural  sequence  of  use  emerges.  Indeed,  the  outputs  of  earlier 
components  can  be  used  as  inputs  for  latter  components.  Although  each  component  could 
be  used  independently  from  the  others,  the  natural  sequence  may  be  beneficial  by  itself  in 
terms  of  support  to  J5  integrated  teams. 

The  pattern  of  results,  observations  and  SME  comments  obtained  from  the  experiments 
and  workshops  suggest  that  a  complete  training  in  the  whole  12om  method  prior  to 
starting  such  an  exercise  would  allow  a  more  effective  use  of  the  methodology  components. 
Consequently,  the  process  involves  training  for  all  components  to  efficiently  show  how  to 
fully  exploit  each  component. 

Moreover,  because  of  the  potential  complexity  in  using  some  of  the  components,  the  12om 
process  ideally  involves  the  inclusion  of  an  interagency  planning  specialist/facilitator  (with 
modeling  expertise)  within  the  team.  In  particular,  while  the  core  features  of  the  IMAGE  v3 
tool  are  deemed  valuable  for  most  users,  the  more  complex  ones,  using  the  specialized 
features  require  a  particularly  well-designed  training  package  or  to  be  supported  by  a 
specialized  facilitator.  One  additional  potential  advantage  of  having  an  interagency 
planning  specialist  within  the  team  is  to  ensure  that  one  team  member  is  not  associated  to 
a  specific  LOO.  Having  such  a  team  member  with  no  a  priori  agenda  may  also  help  to  ensure 
a  more  balanced  consideration  of  factors.  Finally,  one  of  the  key  roles  of  the 
specialist/facilitator  could  be  to  actively  gauge  and  support  collaborative  understanding  - 
particularly  using  communication  reflection  techniques  such  as  mirroring  and 
paraphrasing. 

In  summary,  the  12om  process  involves  three  main  dimensions: 

1.  A  toolbox  approach; 

2.  Training;  and 

3.  Inclusion  of  an  inter-agency  planning  specialist/facilitator. 
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Herein,  we  discuss  how  these  dimensions  can  be  integrated  into  a  comprehensive,  yet 
modular  approach  to  understanding  complex  military  situations:  i.e.,  the  12om  process. 

2.1  A  Toolbox  Approach 

The  version  1.1  of  the  12om  process  is  based  on  a  "toolbox”  approach  (Gigerenzer  & 
Gaissmaier,  2011;  Gigerenzer  &  Selten,  2001;  Gigerenzer,  Todd  &  al.,  1999).  Research  in 
various  domains  suggests  that  the  strength  of  a  toolbox  approach  lies  on  the  fact  that  a 
given  methodology  component  is  not  intrinsically  good  or  bad  in  enhancing  analyst’ 
comprehension,  but  that  its  utility  has  to  be  determined  in  relation  to  the  characteristics 
and  constraints  of  the  task  (Dieckmann  &  Rieskamp,  2007;  Rieskamp  &  Otto,  2006).  Such 
constraints,  for  instance,  are  temporal  pressure  (Karelaia  &  Hogarth,  2008)  and 
uncertainty  (e.g.  Marewski  &  Schooler,  2011).  One  single  component  doesn’t  have  to  be 
applied  to  all  contexts  as  they  can  be  handpicked  at  will  when  relevant.  For  instance,  in 
time  pressured  tasks  where  quick  action  is  required,  one  might  not  have  the  time  to 
develop  an  integrated  representation  of  the  situation  with  a  knowledge  representation  tool, 
however,  they  may  benefit  from  such  a  representation  if  the  situation  does  not  require 
quick  actions. 

Here,  we  first  describe  the  components  integrated  within  the  12om  toolbox  and  describe 
their  specific  process.  After,  we  review  the  conditions  under  which  each  of  the  components 
seems  to  be  the  most  useful.  Finally,  we  recommend  a  natural  sequence  of  when  they 
should  be  used,  relatively  to  their  specific  characteristics  and  environmental  constraints. 

2.1.1  Tools 

Seven  tools  (or  components)  are  integrated  within  the  12om  process:  (1)  WoG  OPP 
handbook,  (2)  Team  building  and  handover  procedure,  (3)  Interactive  common  glossary, 
(4)  IMAGE  v3  (i.e.,  individual  and  collaborative  diagrams),  (5)  Cross-impact  method,  (6)  Op 
Design  tool  and  (7)  Mission  analysis  briefing  template.  Selection  and  integration  of  these 
components  is  based  on  the  assessment  by  DRDC  experts  of  the  critical  results  from  the 
previous  workshops  and  experiments.  Although  these  components  are  integrated  within 
the  overall  12om  process,  they  are  all  characterized  by  an  inner  micro-process.  This  section 
overviews  the  12 om  components  by  providing  a  short  description  and  briefly  explaining 
their  respective  process. 

2.1. 1.1  WoG  OPP  Handbook 

The  WoG  OPP  handbook  is  a  small  document  destined  to  the  members  of  joint  civil-military 
planning  endeavors,  such  as  WoG  approach  initiatives.  It  comprises  of  a  summary  of  the 
main  phases  of  the  OPP  and  associated  sub-tasks.  The  purpose  of  the  handbook  is  to  inform 
joint  civil-military  teams  about  what  is  expected  from  them.  The  integration  of  this 
component  is  justified  by  the  observed  discrepancies  on  the  understanding  of  the  OPP 
across  team  members,  especially  between  civil  and  military  members.  The  lack  of 
understanding  of  the  OPP  by  some  of  the  civilian  members  may  be  responsible  for  their  low 
implication  during  the  OPP. 
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The  process  associated  with  this  specific  component  is  to  distribute  the  handbook  to  all 
members  of  the  planning  team  prior  to  deployment.  The  handbook  is  expected  to  serve  as 
reference  material;  so  each  team  member  can  consult  the  handbook  when  needed. 

2.1. 1.2  Team  Building  and  Handover  Procedure 

Team  building  and  handover  procedure  consists  of  a  set  of  activities  aiming  to  foster 
knowledge  about  the  team’s  goal  and  objectives,  team  members’  expertise  and  specific 
knowledge,  and  about  the  process  that  will  be  followed  by  the  team.  Information  captured 
during  this  set  of  activities  is  recorded  on  a  shared  drive  to  facilitate  handover.  The 
integration  of  this  methodology  component  within  12om  is  justified  in  part  by  the  lack  of 
cohesion  between  members  of  the  team  (especially  across  agencies)  which  has  been 
outlined  by  the  SMEs  during  the  series  of  workshops. 

The  process  involved  with  this  component  is  set  in  two  phases.  Both  phases  engage  the  J5 
(or  other  previously  identified  exercise  lead)  to  lead  a  round  table  discussion  aiming  to 
collect  and  share  information  about  the  members  of  the  team.  The  first  phase  occurs  before 
the  initial  mission  analysis  brief.  Its  objective  is  to  capture  general  knowledge  of  the  team’s 
member.  Such  knowledge  consists  of  information  about  previous  experience  of  team 
members  in  similar  contexts,  expected  departure  date,  and  tool-related  skills,  for  instance. 
The  second  phase,  occurring  after  the  initial  mission  analysis  brief,  aims  to  develop  a 
common  picture  of  the  team’s  goal  and  of  the  process  that  will  be  carried  out  to  achieve  this 
goal.  A  secretary  captures  as  many  of  the  information  possible  in  a  provided  excel  file 
template.  This  file  is  not  shown  to  participants  during  the  exercise  as  it  is  potentially  too 
distracting.  The  file  is  placed  on  a  shared  drive  and  each  participant  is  responsible  for 
checking,  reviewing  and  updating  his/her  own  information  during  the  mission. 

When  turnover  of  personnel  is  predictable,  J5  outlines  the  desirable  procedures  for 
preparation  and  handling  of  personnel  turnover  as  part  of  the  process.  This  process 
involves  the  following  element.  Team  members  are  asked  to  notify  the  J5  when  their 
departure  date  is  known.  Team  members  should  establish,  when  possible,  communication 
between  incoming  and  departing  team  members  as  soon  as  possible.  In  order  to  develop 
incoming  member’s  understanding  of  the  mission,  the  departing  member  should  include 
the  incoming  member  in  all  his/her  relevant  correspondence.  Finally,  departing  member 
should  stay  in  touch  with  the  replacement  even  after  his/her  own  departure  from  the  team. 

This  process  is  greatly  facilitated  by  higher  levels  of  command  is  there  is  a  consideration 
for  early  identification  of  replacements,  planning  for  overlapping  periods  between 
incoming  and  departing  team  members,  and  active  support  for  facilitating  communications 
between  them. 

2.1. 1.3  Interactive  Common  Glossary 

The  interactive  common  glossary  is  a  tool  that  centralizes  and  defines  situation-related 
terms  and  acronyms.  Its  purpose  is  to  foster  the  use  of  a  common  language  during  the  OPP. 
The  integration  of  this  methodology  component  within  12om  is  justified  by  the  observed 
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discrepancies  observed  between  individuals  and/or  agencies  which  often  lead  to 
miscomprehension  and  lack  of  collaboration. 

The  process  involved  with  this  component  is  unique  as  it  would  ideally  always  be  on-going, 
even  before  mission  analysis  even  starts.  The  common  glossary  should  be  distributed 
across  relevant  organizations  so  that  they  can  consult  and  modify  it  at  will.  This  would 
raise  acceptance  and  relevance  of  the  component  across  organizations.  During  a  planning 
mission,  since  the  component  is  mostly  reference  material,  it  should  be  used  when  needed, 
individually. 

2.1. 1.4  IMAGE  v3 

IMAGE  v3  is  a  set  of  tools  and  techniques  that  aims  to  facilitate  the  integration  of  different 
perspectives  on  a  situation  into  a  single  visual  representation.  Depending  on  the  level  of 
achieved  formalism,  this  representation  can  be  a  sole  description  of  the  situation  or  a 
computational  model  of  a  complex  system.  The  integration  of  this  component  within  12om 
is  justified  by  the  difficulties  observed  in  comprehensively  representing  complex  situations 
in  the  context  of  OPP.  Moreover,  this  component  seeks  to  foster  the  integration  of  multiple 
perspectives,  which  is  also  frequently  lacking  in  this  context 

Although  the  final  decision  should  always  be  the  prerogative  of  the  leader  of  the  WoG  team, 
we  suggest,  based  on  feedback  from  participants  and  our  observations,  that  planning  teams 
engaged  in  the  creation  of  conceptual  diagrams  should  begin  with  the  creation  of  a 
collaborative  diagram  rather  than  individual  diagrams. 

However,  if  individual  diagrams  are  created,  the  process  should  include  specific  tasks. 
Since  the  intention  of  individual  conceptual  diagrams  is  to  ultimately  integrate  them  into  a 
common  representation,  an  initial  coordination  about  the  main  parameters  (e.g.,  scale, 
level  of  details)  and  major  concepts/issues  would  give  a  good  starting  point  for  the 
individual  work  and  significantly  facilitate  future  integration.  Because  one  of  the  main 
issues  associated  with  this  component  is  the  time  requirement,  the  lead  of  the  mission  (e.g., 
J5)  should  be  aware  of  the  costs  of  using  IMAGE  v3  (in  terms  of  time  needed  to  carry  out 
the  analysis)  in  order  to  make  an  informed  decision  about  its  adoption  or  not  in  the  context 
of  the  mission. 

2.1. 1.5  Cross-Impact  Method 

The  cross-impact  method  is  a  structured  way  to  analyse  a  situation  and  increase  awareness 
of  critical  interactions.  It  aims  to  improve  shared  awareness  of  the  multiple  factors 
involved,  to  make  salient  key  divergences  in  understanding  among  a  team  and  to  minimize 
tunnel  vision  by  promoting  a  comprehensive  consideration  of  factors’  direct  and  indirect 
effects.  Moreover,  in  the  suggested  implementation,  the  cross-impact  method  also 
promotes  option  awareness,  that  is  what  can  be  done  to  improve  a  particular  variable 
within  the  system.  Despite  lower  ratings  from  SMEs  concerning  the  feasibility  of 
implementing  such  a  component  within  the  OPP,  the  integration  of  this  component  within 
12om  is  justified  mainly  by  the  need  for  a  method  that  can  bridge  the  gap  between 
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situation  representation  and  initial  course  of  action  development.  Accordingly,  the  method 
has  undergone  several  modifications  in  response  to  SMEs  comments  and  suggestions.  For 
instance,  the  size  of  the  matrix  is  now  limited  to  9X9  in  order  to  reduce  its  complexity  and 
the  time  required  to  perform  the  analysis.  A  more  easily  understandable  graphical  output  is 
now  also  generated  to  facilitate  the  interpretation  of  the  method  and  its  use  as  a 
communication  tool. 

The  CIM  follows  a  three-step  approach  (two  mandatory,  one  optional).  The  first  step  aims 
to  identify  and  select  the  elements  to  be  included  in  the  analysis.  The  second  step  is  to  map 
these  elements  against  each  other  and  determine  their  mutual  influences.  Finally,  the  third 
step  is  to  define  possible  interventions  and  assess  their  impact. 

Step  1  -  Variable  identification.  This  step  is  broken  down  further  more  into  two  sub-steps: 
select  variables  and  specify  subgoals.  Subgoals  are  critical  for  the  analysis  because  they 
allow  assessing  how  different  effects  relate  to  the  mission  objective.  Variables  that  are  not 
subgoals  are  influencing  factors.  Influencing  factors  directly  or  indirectly  impact  subgoals. 
Note  that  subgoals  can  be  influencing  factors  as  well,  butthe  label  "subgoal"  takes  priority. 

Sub-Step  1.1  -  Variable  selection.  In  the  context  of  a  planning  team,  the  process  for 
selecting  variables  is  democratic,  each  team  member  voting  for  the  desired 
variables. 

Sub-Step  1.2  -  Specify  subgoals  directionality.  You  may  seek  to  maximize  or  minimize 
a  given  subgoal.  For  instance,  “crime”  is  not  a  subgoal  per  se,  but  "minimizing  crime” 
might  be  one.  The  second  step  of  variable  identification  aims  precisely  at  specifying 
the  direction  of  the  subgoals  (which  is  critical  for  the  analysis).  Here,  there  is  no 
specified  process  on  how  to  establish  goal  directionality  as  it  depends  heavily  on  the 
context  (e.g.,  the  goals  and  subgoals  maybe  specified  by  higher  levels  of  command). 

Step  2  -  Influence  mapping.  This  step  is  also  broken  down  into  two  sub-steps.  The  first  step 
is  to  actually  perform  the  mapping  of  the  influences  between  the  selected  variables.  The 
second  step  is  to  assess  the  output  generated  by  the  mapping. 

Sub-Step  2.1  -  Cross  impact  mapping.  One  row  at  a  time,  team  members  must  map 
the  effects  of  the  variable  on  the  left  on  the  variables  in  each  column,  if  applicable. 
This  process  should  be  supported  by  a  facilitator.  It  is  critical  that  the  mapping 
reflects  direct  influences  only  rather  than  covariation  of  variables. 

Sub-Step  2.2.  -  Review  output.  The  output  will  order  the  most  influencing  variables 
(positive  and  negative)  with  regard  to  priory  specified  subgoals.  This  information 
may  be  critical  for  sequencing  decisive  points  when  using  the  Op  Design  tool.  A 
decisive  point  is  a  specific  key  event,  critical  factor,  or  function  that,  when  acted 
upon,  allows  commanders  to  gain  a  marked  advantage  over  an  adversary  or 
contribute  materially  to  achieving  success.  Decisive  Points  may  exist  in  time,  space 
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or  in  the  information  environment.  Since  decisive  points  need  to  be  ordered  into 
lines  of  operations  (i.e.,  because  limited  manpower/resources  require  actions  to  be 
performed  sequentially),  the  selection  of  which  action  to  do  first  can  be  guided  by 
the  expected  payoff  of  the  CIM  (doing  high  payoff  actions  first  will  "generate  more 
interest”  over  time).  There  are  other  considerations  for  decisive  point  analysis,  such 
as  sequential  dependencies,  or  potential  conflicts/synergies  when  synchronising 
some  decisive  points. 

Step  3  -  Intervention  matrix.  This  optional  step  is  also  divided  in  two  sub-steps: 
intervention  mapping  and  output  review.  The  purpose  of  the  intervention  matrix  is  to 
allow  for  a  comparison  of  up  to  nine  potential  interventions.  Cost-benefit  assessment  of  the 
direct  impacts  of  each  intervention  on  subgoals  (1st  order  effects),  plus  the  overall  impacts 
on  subgoals  (last  column  of  the  cross  impact  matrix)  of  the  variables  influenced  by  the 
intervention  (i.e.,  2nd  and  3rd  order  effects).  Note  that  reducing  an  unfavorable  variable 
whose  global  impact  is  -5  counts  as  a  favorable  effect  of  5. 

Sub-Step  3.1  -  Intervention  mapping.  One  row  at  a  time,  team  members  must  map 
the  effects  of  the  possible  interventions  on  the  variables,  if  applicable.  This  process 
should  be  supported  by  a  facilitator.  It  is  critical  that  the  mapping  reflects  direct 
influences  only  rather  than  covariation  of  variables. 

Sub-Step  3.2.  -  Review  output.  Similarly  to  step-2  output,  step-3  output  graph 
provides  guidance  with  regards  to  decisive  point  sequencing,  assuming  that  an 
intervention  can  be  readily  associated  to  a  decisive  point.  In  addition  to  the  order  of 
decisive  points,  this  assessment  may  motivate  the  addition/removal  of  a  decisive 
point  (in  the  case  of  interventions  with  counter-intuitively  favorable/unfavorable 
systemic  impacts). 

2.1. 1.6  Op  Design  Tool 

The  Op  Design  tool  aims  to  support  planners  in  sequencing  decisive  points  into  lines  of 
operations  and  to  identify  operational  phases  with  their  associated  objectives  and  tasks.  It 
provides  the  grounds  to  initiate  the  thinking  required  to  identify  possible  branch  plans 
and/or  sequel  plans  where  transition  conditions  are  desired.  The  integration  of  this 
component  within  12om  is  justified  by  the  need  for  initial  OP  Design  during  the  orientation 
phase  of  the  OPP  and  the  limited  capabilities  of  the  other  components  to  satisfy  this  need. 

The  process  of  decisive  point  analysis  using  the  Op  Design  tool  is  composed  of  six  steps: 

(a)  Step  1  -  Identify  decisive  points, 

(b)  Step  2  -  Sequence  decisive  points, 

(c)  Step  3  -  Identify  phases, 

(d)  Step  4  -  Identify  objectives, 

(e)  Step  5  -  Identify  tasks,  and 

(f)  Step  6  -  Identify  branch,  think  about  sequel. 


31  March  2014 


19 


Version  1.1 


12om  Process 


ci 


£ 


THALES 


2.1. 1.7  Integrated  Mission  Analysis  Briefing  Template 

The  integrated  MA  briefing  template  intends  to  help  clarify  the  nature  of  the  output 
required  by  each  team  member  and  to  reduce  formatting  work  for  the  mission  analysis 
brief.  The  integration  of  this  component  in  the  12om  toolbox  is  justified  by  the  fact  that  a 
pure  military  template  for  MA  briefing  may  not  capture  aspects  relevant  to  other  agencies 
like  CIDAand  DFAIT. 

The  MA  briefing  template  does  not  involve  a  specific  process  per  se.  The  only  "process" 
would  be  that  the  template  should  be  reviewed  and  filled  out  by  the  team  collaboratively 
rather  than  by  separating  the  job  across  individuals.  This  would  allow  for  a  better 
integration  of  the  different  lines  of  operations  involved  in  the  planning  endeavour. 

2.1 .2  The  Right  Tool  for  the  Right  Task:  Linking  Tools  to  the  Process 

As  mentioned  earlier,  none  of  the  components  integrated  within  the  12om  methodology  is 
mandatory  when  performing  OPP.  Each  component  can  be  hand-picked  in  an  adaptive 
manner,  when  relevant.  Utility  of  each  component  will  vary  given  the  characteristics  of  the 
tasks  to  be  carried  out.  Herein,  we  highlight  the  favourable  conditions  of  the  use  of  each 
component,  in  particular  during  initiation  and  orientation  phases.  This  is  done  by  making 
explicit  the  conditions  under  which  the  characteristics  of  the  component  fit  with  the 
characteristics  of  the  task.  This  proposition  is  based  specifically  on  the  evaluations  of  the 
components  by  the  SMEs  and  observations  made  during  experimentations. 

In  addition  to  the  description  of  the  tools,  the  present  section  provides  means  to  link  the 
12om  components  with  the  tasks  to  be  carried  out  and  consequently  specify  the  process  on 
step  further.  The  goal  here  is  to  escape  from  the  tool-specific  process  by  considering  the 
process  of  the  whole  12om  toolbox.  The  conditions  stated  below  are  the  considerations 
that  allow  a  planner  to  choose  which  components  he  should  use  for  his  work.  Moreover,  the 
conditions  specified  below  are  generic  enough  to  be  applied  to  a  wide  variety  of  contexts, 
including  but  not  reserved  to  CFOPP. 

2.1.2.1  WoG  OPP  Handbook 

The  WoG  OPP  handbook  would  ideally  be  handed  out  prior  to  actual  work.  Conditions 
favourable  to  the  efficient  use  of  WoG  OPP  handbookare: 

•  Knowledge  of  the  staff  involved  in  the  mission  -  If  the  staff  is  unknown  before 
the  mission,  it  won’t  be  possible  to  distribute  the  handbook; 

•  Guideline  -  The  handbook  describes  the  original  OPP.  Any  deviation  from  it  in  the 
operational  context  will  not  be  captured  by  the  handbook.  Therefore,  the  handbook 
should  be  considered  as  a  flexible  guideline  rather  than  a  prescriptive  method;  and 

•  Low  time  pressure  -  If  the  mission  requires  quick  deployment  and  actions,  the 
time  available  to  read  the  handbook  may  be  very  limited  and  therefore  render  its 
use  irrelevant 
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2.1. 2.2  Team  Building  and  Handover  Procedure 

The  team  building  and  handover  procedure  should  be  completed  at  the  very  beginning  of 
the  mission  [or  even  before  according  to  some  SMEs).  Conditions  in  which  this  procedure 
would  be  the  most  useful  involves: 

•  Non-conflicting  agendas  -  The  application  of  this  procedure  will  be  limited  if  team 
members  have  conflicting  agendas,  and  therefore  cannot  attend  the  meeting.  Ideally, 
this  procedure  would  involve  all  J5  integrated  team  members; 

•  Interoperable  systems  -  This  procedure  would  be  more  beneficial  if  all  staff 
members  share  the  same  working  environment.  This  would  be  necessary  to  easily 
access  and  update  the  information  collected  during  the  procedure; 

•  Low  time  pressure  -  The  use  this  method  may  be  time  consuming.  Therefore,  if  the 
mission  requires  quick  actions,  it  may  not  be  possible  to  use  this  component 
effectively;  and 

•  Uncertainty  -  This  procedure  would  reduce  uncertainty  associated  with  the 
beginning  of  a  mission,  especially  uncertainty  pertaining  to  the  mission’s  objective 
and  the  OPP  process. 

2.1. 2.3  Interactive  Common  Glossary 

The  interactive  common  glossary  can  be  useful  throughout  the  OPP.  Conditions  in  which 
the  common  glossary  would  be  the  most  useful  involves: 

•  Multi-agency  -  Terms  are  more  heterogeneous  across  agencies  than  within 
agencies.  Therefore,  a  multi-agency  team  may  benefit  more  of  a  glossary  than  a 
single-agency  team; 

•  Interoperable  systems  -  The  interactive  common  glossary  would  be  effective  only 
if  accessible  to  all  team  members;  and 

•  On-going  consultation  and  modifications  -  The  general  glossary  requires  on¬ 
going  consultation  and  modifications  between  departments  to  build  visibility  and 
acceptance. 

2.1. 2.4  IMAGE  v3 

IMAGE  v3  is  used  for  the  creation  of  conceptual  diagrams.  Conceptual  diagrams  are  useful 
for  integrating  different  perspectives  into  a  single  representation,  and  describing  the 
situation  either  individually  or  collectively.  The  conditions  in  which  IMAGE  v3  would  be  the 
most  useful  involves: 

•  Multiple  agencies  -  Collaborative  knowledge  representation  would  facilitate  the 
identification  of  differences  between  LOOs.  Moreover,  the  use  of  representation 
fosters  collaboration  between  team  members.  Therefore,  the  use  of  this  component 
is  adapted  to  a  multi-agency  mission; 

•  Common  terminology  -  Divergence  in  vocabulary  terms  when  using  collaborative 
representation  tool  maybe  an  issue.  Henceforth,  the  use  of  such  tool  would  benefit 
from  the  use  of  a  common  glossary  tool  in  parallel,  such  as  the  one  suggested  in  the 
actual  12om  toolbox; 
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•  Complex  situations  -  The  use  of  such  representation  helps  in  making  sense  of 
complex  situations.  IMAGE  v3  is  useful  for  identifying  additional  factors  and 
influences,  and  stimulating  reflection  on  key  assumptions; 

•  Low  time  pressure  -  The  use  of  IMAGE  v3  may  be  time  consuming.  Therefore,  if  the 
mission  requires  quick  actions,  it  may  not  be  possible  to  use  this  component 
effectively.  Some  SMEs  even  suggested  that  this  component  could  be  used  before 
mission  analysis; 

•  Work  in  progress  -  IMAGE  v3  is  a  great  "thinking”  tool,  but  not  so  great  for  briefing 
purposes,  because  the  resulting  product  -  the  diagram  -  can  easily  get  too  complex 
impeding  its  readability.  Therefore,  these  tools  should  be  used  mainly  during  the 
review  of  the  situation  and  factors  analysis;  and 

•  Specialized  knowledge  -  The  complexity  of  the  tools  makes  it  difficult  for  novices 
to  use.  Therefore,  the  presence  of  a  team  member  specialized  in  the  use  of  such  tools 
would  be  a  favourable  condition. 

2.1. 2.5  Cross-Impact  Method 

Cross-impact  method  aims  to  improve  shared  awareness  of  the  multiple  factors  involved, 
to  make  salient  key  divergences  in  understanding  among  a  team  and  to  minimize  tunnel 
vision  by  promoting  a  comprehensive  consideration  of  factors'  direct  and  indirect  effects. 
Based  on  the  information  collected  during  workshops,  the  context  in  which  cross-impact 
method  would  be  the  most  useful  involves: 

•  Complex  situations  -  The  use  of  the  method  helps  in  making  sense  of  complex 
situations  because  it  forces  its  user  to  systematically  evaluate  the  relationships 
between  the  variables  of  a  situation; 

•  Low  time  pressure  -  The  use  this  method  may  be  time  consuming.  Therefore,  if  the 
mission  requires  quick  actions,  it  may  not  be  possible  to  use  this  component 
effectively.  A  first  draft  of  the  CIM  could  be  completed  by  the  specialist  to 
considerably  reduce  time/effort  required  from  the  entire  team,  which  could  then 
refine  that  draft  collaboratively; 

•  Leadership  -  Due  to  the  level  of  training  and  the  time  required  for  completing  the 
CIM  collaboratively,  there  is  a  risk  that  the  tool  would  be  skipped  in  the  absence  of 
leadership  to  use  it.  This  leadership  could  come  from  a  specialist/facilitator  already 
familiar  with  this  component; 

•  Interest  in  general  dynamics  -  This  method  will  help  understand  the  general 
dynamics  involved  in  a  system.  It  won't  help  in  finding  idiosyncrasies  or  micro 
patterns  of  evolution,  e.g.,  it  will  not  help  identifying  who  the  key  leader  is  in  a  local 
insurgency  group,  but  it  may  help  identifying  the  overall  impact  of  the  group  on  the 
security  in  the  region.  This  is  partly  due  to  the  difficulty  to  include  some  types  of 
variables  such  as  terrain,  and  on  the  combinatorial  complexity  associated  with  the 
addition  of  new  factors  (the  analysis  focuses  only  on  the  most  critical  factors); 

•  Course  of  actions  -  The  action-oriented  section  of  the  method  may  be  premature 
for  mission  analysis,  and  may  perhaps  be  more  suited  for  initial  COA  development. 
This  step  is  therefore  optional  for  mission  analysis,  but  can  be  useful  for  Op  Design 
since  is  supports  option  exploration  and  DP  sequencing;  and 
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•  Briefing  -  The  output  of  the  method  could  be  a  nice  addition  for  briefing. 

2.1.2.6  OP  Design  Tool 

The  Op  Design  tool  aims  to  support  the  planners  in  their  brainstorming  to  sequence 
decisive  points  into  lines  of  operations  and  to  identify  operational  phases  with  their 
associated  objectives  and  tasks.  Based  on  the  information  collected  during  workshops,  the 
context  in  which  DP  Analysis  would  be  the  most  useful  involves: 

•  Planning  -  DP  analysis  will  be  mostly  relevant  in  the  OP  Design  phase  of  mission 
analysis; 

•  Multiple  objectives  -  This  analysis  is  very  relevant  to  take  multiple  objectives  into 
account; 

•  Multiple  agencies  -  This  analysis  is  very  relevant  for  synchronising/coordinating 
multiple  LOOs  in  the  achievement  of  multiple  goals; 

•  Time  horizon  -  The  multiple  objectives  and  agencies  should  have  similar  timelines. 
The  effective  use  of  this  component  will  be  more  difficult  if,  for  instance,  one 
organisation  has  a  1-year  horizon  for  its  objective  and  another  has  a  5-year  horizon; 

•  Briefing  -  SMEs  agreed  that  the  analysis  produces  a  useful  output  for  briefing; 

•  Interoperable  systems/procedure  -  The  SMEs  stated  that  the  method  would 
work  best  in  conditions  where  the  interoperability  with  other  systems  would  be 
high.  For  instance,  SMEs  from  CIDA  stated  that  they  approach  situations  differently 
(change  oriented  rather  than  action  oriented)  and  that  it  may  not  fit  with  the 
approach  used  in  decisive  point  analysis; 

•  Specialized  knowledge  -  The  complexity  of  the  tool  makes  it  difficult  for  anyone  to 
use  it  at  hand.  Therefore,  the  presence  of  a  team  member  specialized  in  the  use  of 
the  tool  would  be  a  favourable  condition;  and 

•  Course  of  actions  -  The  component  may  be  premature  early  in  mission  analysis, 
and  may  perhaps  be  more  suited  for  initial  OP  Design  and  COA  development 

2.1 .2.7  Integrated  M  A  Briefing  T emplate 

The  integrated  MA  briefing  template  intends  to  help  clarify  the  nature  of  the  output 
required  by  each  team  member  and  to  reduce  formatting  work  for  the  mission  analysis 
brief.  Based  on  the  information  collected  during  workshops,  the  context  in  which 
integrated  MA  briefing  template  would  be  the  most  useful  involves: 

•  Multiple  agencies  -  The  template  is  designed  to  facilitate  the  integration  of 
multiple  perspectives  on  the  situation; 

•  High  time  pressure  -  The  template  facilitates  putting  together  the  evidence  and 
information  collected  by  the  team  into  a  comprehensive  frame.  It  may  help  structure 
teamwork  and  consequently,  increase  work  efficiency.  Therefore,  the  template 
should  help  the  team  to  produce  an  output  faster;  and 

•  Uncertainty  -  The  integrated  template  clearly  states  the  required  pieces  of 
information  required  for  the  MA  template.  Consequently,  it  may  help  decrease 
uncertainty  associated  with  the  nature  of  the  deliverables  requested  from  the  team; 
and 

•  Collaborative  effort  -  The  process  for  preparing  briefing  should  be  collaborative. 
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2.1 .3  12om  in  the  Context  of  OPP 

The  conditions  listed  above  are  useful  for  guiding  the  selection  and  use  of  the  appropriate 
component  in  the  context  of  a  specific  planning  endeavour.  In  addition,  a  functional  gap 
analysis  (See  Appendix  A)  revealed  that  some  components  are  general  whereas  others  are 
specific.  Together,  these  analyses  show  that  the  components  should  not  be  used  without 
considering  their  strengths  and  weaknesses. 

Other  considerations  should  be  taken  into  account  when  sequencing  the  use  of  the 
components:  the  leverage  effect  of  the  outputs,  specifically.  For  instance,  the  output  of 
collaborative  knowledge  representation  can  feed  the  cross-impact  method,  and  the  output 
of  the  cross-impact  method  can  be  integrated  in  the  MA  briefing  template.  Here,  we  list 
how  some  of  the  output  of  the  12om  components  can  be  re-used  into  another  component 
and  provide  a  leverage  effect 

1.  The  WoG  OPP  handbook  does  not  generate  any  output  per  se,  however,  the 
knowledge  gained  from  it  is  useful  throughout  the  whole  OPP.  Consequently,  one 
could  say  that  the  WoG  OPP  handbook  indirectly  facilitates  the  process  of  all 
components. 

2.  The  objectives  specified  during  the  second  phase  of  the  team  building  procedure 
could  serve  as  a  basis  for  the  sub-goals  identified  in  the  cross  impact  method  and/or 
the  Op  Design  tool.  Although  this  is  not  a  direct  use  of  the  output,  it  could 
nonetheless  facilitate  the  process. 

3.  The  terms  defined  in  the  common  glossary  could  be  used  in  all  other  components.  It 
could  be  used  as  reference  material  to  inform  a  user  about  the  meaning  of  a 
term/acronym  in  a  document.  It  could  also  be  used  as  a  form  of  constraint  when 
using  other  components  to  ensure  the  use  of  common  terms  by  the  team,  notably  in 
IMAGE  v3,  the  cross  impact  method,  the  Op  Design  tool,  and  the  mission  analysis 
briefing  template. 

4.  IMAGE  v3  outputs  come  in  many  formats  which  can  all  be  used  for  different 
purposes.  The  graphical  outputs  generated  could  be  integrated  within  the  mission 
analysis  briefing  template,  depending  on  the  commander’s  will.  It  may  serve  as  a 
support  for  the  presenter  during  the  brief.  The  table  format  outputs  generated  by 
IMAGE  v3  can  be  imported  directly  into  the  cross  impact  method  and  the  Op  Design 
tool.  The  table  format  outputs  can  greatly  facilitate  the  variable  selection  process  of 
the  cross-impact  method  and  the  five  first  steps  of  the  decisive  point  analysis  using 
the  Op  Design  tool. 

5.  The  cross  impact  method  also  generates  both  a  graphical  and  a  table  format  output. 
The  graphical  outputs  of  the  cross  impact  method  may  be  integrated  within  the 
mission  analysis  briefing  template  to  support  the  presenter  during  the  mission 
analysis  brief.  On  the  other  hand,  the  table  format  output  could  be  imported  back 
into  IMAGE  v3  and  provide  basis  for  the  development  of  a  new  knowledge 
representation. 

6.  The  Op  Design  tool  generates  a  very  useful  graphical  representation  that  shows 
inter-relations  across  the  lines  of  operations  with  regards  to  the  mission’s 
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objectives.  This  graphical  representation  could  be  copy  pasted  into  the  mission 
analysis  briefing  template  and  be  used  for  supporting  the  presenter  during  the  brief. 

7.  The  integrated  mission  analysis  briefing  template  is  expected  to  provide  a  structure 
to  the  mission  analysis  brief.  Consequently,  this  structure  could  be  used  during  the 
actual  mission  analysis  brief. 

By  considering  [a)  the  mapping  between  components  and  the  OPP  tasks,  combined  with 
(b)  the  functional  gap  analysis  conducted  during  the  LOE#2  (see  Appendix  A],  and  (c)  the 
leverage  that  can  be  obtained  by  building  upon  the  outputs  of  each  components,  the 
sequencing  of  the  components  during  OPP  results  into  a  "natural”  sequence  of  introduction 
and  usage  of  the  12om  methodology  components.  This  natural  sequence  is  shown  in  Figure 
5  and  Figure  6. 


2.2  Training 

The  second  dimension  of  the  12om  process  relates  to  training.  Training  is  critical  in  the 
context  of  the  12om  methodology  and  has  to  be  emphasised  as  an  important  dimension  of 
the  12om  process.  A  training  package  is  available  elsewhere  (see  ATT7  deliverable 
"Training  Modification  Recommendations")  and  serves  as  a  guide  through  the  training 
process.  The  training  is  divided  into  three  steps: 

1.  Contextualization, 

2.  Introductory  training,  and 

3.  Specific  training  and  exercises. 

2.2.1  Contextualization 

This  step  mainly  describes  the  objectives  (i.e.,  supporting  WoG  teams  in  the  comprehension 
of  complex  situations),  the  scope  (i.e.,  initiation  and  orientation  phases  of  the  CFOPP),  and 
the  target  areas  of  the  12om  methodology  (i.e.,  shared  understanding,  collaboration,  and 
integration  of  different  perspectives).  It  is  a  high  level  description  of  the  context  of 
application  of  the  methodology. 

2.2.2  Introductory  Training 

This  step  describes  the  toolbox  approach  of  the  12om  methodology.  It  stresses  that  a 
particular  component  is  not  good  or  bad  per  se,  but  that  its  effectiveness  is  determined  by 
the  characteristics  of  the  tasks  to  be  carried  out.  Afterward,  the  training  presents  all  the 
components  of  the  12om  toolbox  in  terms  of  purposes  and  other  key  characteristics.  For 
instance,  it  enumerates  how  the  outputs  of  each  component  can  be  useful  to  the  other  ones. 
Finally,  the  natural  sequence  emerging  from  the  mapping  between  tasks  characteristics 
and  favourable  conditions  of  use  is  presented. 
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Figure  5:  Suggested  use  of  the  12om  methodology  components  during  the  initiation  phase  of  the  OPP. 
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3  Conclusion 

Hrychuk  and  Gizewski  (2010)  describe  the  Comprehensive  Approach  as  seeing  "diplomatic, 
defence,  development  and  commercial  resources,  aligned  with  those  of  numerous  other 
agencies,  coordinated  through  an  integrated  campaign  plan  and  applied  in  areas  of 
operations  as  needed"  (p.  2-1).  According  to  Hrychuck  and  Gizawski,  this  approach  is  based 
on  four  basic  tenants: 

•  Proactive  engagement  between  actors; 

•  Shared  understanding; 

•  Outcome-based  thinking  (i.e.,  in  relation  to  a  desired-end  state);  and 

•  Collaborative  working. 

The  12om  methodology  clearly  addresses  each  of  these  four  elements.  The  team  building 
procedure  seeks  to  support  proactive  engagement  between  actors,  the  WoG  OPP  handbook 
and  interactive  common  glossary  seek  to  support  collaborative  working,  the  IMAGE  v3 
component  seeks  to  support  shared  understanding,  and  finally  the  OP  Design  component 
and  cross-impact  method  seek  to  support  outcome-based  thinking. 

Schmorrow  and  Boiney  (2013)  explain  that  goals  can  conflict  at  times  within  government 
departments  and  agencies,  potentially  hindering  the  success  of  WoG  initiatives.  Different 
planning  time  horizons  also  contribute  to  this  problem.  They  add  that  "A  common  planning 
process  or  tool,  which  could  be  used  by  various  government  departments  and  agencies,  as 
well  as  NGOs,  would  help"  (p.  10-3).  Schmorrow  and  Boiney  (2013)  also  mention  several 
R&D  priorities  to  help  succeed  in  the  implementation  of  a  comprehensive  approach, 
including  the  following  areas  that  were  well  covered  by  the  12om  project: 

•  Methods  to  collect,  integrate  and  visualize  socio-cultural  information  necessary  for 
supporting  a  comprehensive  approach; 

•  Multi-agency  modeling,  simulation,  and  experimentation  to  develop  a 
comprehensive  approach  operational  concept; 

•  Tools,  methods  and  techniques  to  support  force  synchronization;  and 

•  Tools  for  forecasting  first-  to  third-order  effects  to  support  intelligence,  course  of 
action  (COA)  development,  and  decision  making. 

Taking  an  empirical  approach  (LOEs  and  SME  workshops  with  data  collection)  was  a 
valuable  way  to  inform  the  development  of  the  12om  process  and  tools  and  to  ensure  that 
the  solutions  were  indeed  effective  in  supporting  the  interagency  planning  process.  One 
reason  for  this  is  that  best  practices  for  an  effective  comprehensive  approach  are  still 
unclear.  In  light  of  this,  Smith,  Pavlovic,  Eustace,  Fortier,  Boiney,  Vogelaar,  Lawrence, 
Shockey,  McAuliffe  and  Roy  (2013)  highlight  the  importance  of  empirical  work  using  data 
collected  from  human-in-the-loop  experiments  to  identify  such  best  practices.  One  possible 
criticism  of  the  present  work  is  that  it  relied  strongly  on  subjective  ratings  from  SMEs. 
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While  this  information  is  of  great  value,  subjective  judgments  can  be  prone  to  biases. 
Future  work  would  indeed  benefit  from  placing  a  greater  effort  on  collecting  objective 
process  and  outcome  measures. 

3.1  Limitations  and  Mitigation 

Despite  the  extensive  work  that  was  performed  in  the  development  of  the  12om 
methodology,  there  are  some  known  limitations  that  should  be  made  aware  of.  This  section 
lists  the  main  identified  limitations  of  the  12om  methodology  and  provides  potential 
mitigating  interventions  when  possible. 

The  first  main  limitation  concerns  a  subset  of  the  components  which  are  very  specific  to 
the  OPP.  This  is  the  case  for  the  WoG  OPP  handbook,  the  integrated  mission  analysis 
briefing  template,  and  the  Op  Design  tool  (although  the  latter  is  slightly  less  specific  to  OPP, 
it  remains  strongly  tied  to  military  mission  planning).  This,  of  course,  is  only  a  limitation  if 
one  wishes  to  apply  the  12om  methodology  in  other  contexts.  However,  there  are  several 
advantages  to  being  specific,  such  as  specialisation.  The  main  mitigation  intervention  for 
this  is  to  provide  generic  components  as  well  within  the  12om  toolbox  (which  is  already 
the  case)  so  that  the  planning  teams  still  have  tools  to  choose  from  when  outside  the 
context  of  OPP.  Another  potential  mitigating  intervention  would  be  to  develop  more 
generic  versions  of  these  components. 

The  second  limitation  is  the  time  and  effort  required  for  applying  the  12om  methodology. 
Although  the  methodology  is  intended  to  be  modular,  any  component  integrated  within  the 
planning  process  represents  (or  is  at  least  perceived  as)  an  overhead  in  comparison  to 
what  is  already  usually  done.  Several  mitigating  measures  could  however  be  done  to 
overcome  this  limitation.  Training  of  the  12om  methodology  could  be  integrated  to  the  CF 
Officer  curriculum  and  would  be  consequently  become  the  "usual".  Moreover,  as  previously 
discussed,  the  development  of  a  more  efficient  training  could  reduce  both  time  for  training 
and  using  the  components  in  operational  contexts.  Finally,  also  as  discussed  before,  the 
integration  of  a  12om  facilitator  within  a  planning  team  could  also  help  in  reducing  the 
time  and  effort  required  for  employing  the  methodology. 

3.2  Next  Steps 

Future  work  could  be  centered  on  the  mitigating  interventions  mentioned  above:  (a) 
develop  more  generic  components,  (b)  develop  a  more  efficient  training  process,  and  (c) 
develop  and  integrate  a  new  facilitator  role. 

3.2.1  Generic  Components 

Although  the  context  used  for  demonstration,  development  and  validation  purposes  was 
the  OPP,  the  12om  project  seeks  to  develop  a  methodology  (i.e.,  process  and  tools)  to 
improve  the  understanding  of  a  complex  situation  by  a  multidisciplinary,  government-wide 
team.  It  is  not  intended  specifically  only  for  the  OPP.  From  that  perspective,  some  of  the 
components  may  be  too  specific  to  the  context  of  OPP  and  may  consequently  benefit  from  a 
decrease  in  terms  of  specificity.  For  instance,  the  OPP  handbook  could  be  reviewed  as  a 
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"best-practice  in  planning  handbook".  The  same  logic  could  apply  to  the  briefing  template 
as  well.  It  might  also  be  worth  investigating  the  terms  employed  in  the  Op  Design  tool  to 
see  how  they  could  be  "translated”  into  more  civilian  terms  such  as  the  ones  used  in 
results-based  management,  for  instance. 

3.2.2  Develop  More  Efficient  Training 

Training  time  and  effort  was  a  major  concern  for  the  participants  at  the  workshops  and 
experimentation  (see  L0E#2  results).  Although  some  of  the  components  are  relatively 
simple  and  intuitive,  other  components  may  require  long  training  time  and  effort, 
especially  in  order  to  use  the  more  complex  features.  This  is  the  case,  for  instance,  of  the 
IMAGE  v3  component  for  which  the  basic  individual  knowledge  representation  features 
were  deemed  relatively  easy  to  use  but  for  which  the  graph  querying  feature  necessitated 
more  training. 

We  propose  that  a  "small  scale”  serious  game  would  be  an  optimal  way  for  team  members 
to  practice  the  methodology  within  a  short  time  frame,  and  to  get  a  sense  of  the  significant 
potential  of  the  12om  methodology  components  and  their  synergies.  A  serious  game  helps 
to  gain  experience  without  any  negative  impact  on  reality.  It  enables  the  trainee  to  carry 
out  actions  with  the  actual  12om  components,  without  fear  and  risks  he  could  not  carry  out 
in  reality,  because  their  analysis  would  be  too  slow,  too  fast,  too  expensive,  too  complex  or 
too  dangerous  (Reuters  et  al.,  2009).  Moreover,  the  serious  gaming  approach  gives  the 
trainee  a  coherent  training  example  throughout  the  whole  12om  process. 

Serious  game  technology  is  an  effective  means  to  meet  a  wide  variety  of  training 
requirements  and  is  particularly  well  suited  to  developing  the  cognitive  skills  necessary  to 
turn  a  team  of  experts  into  an  expert  team  (Roman  &  Brown,  2008).  The  learning  process  is 
composed  of  four  main  steps: 

1.  Concrete  experience; 

2.  Observation  of  and  reflection  on  that  experience; 

3.  Formation  of  abstract  concepts  based  upon  the  reflection; 

4.  Testing  the  new  concepts. 

Based  on  this  model,  serious  games  can  be  designed  to  facilitate  iteratively  proceeding  all 
of  these  4  steps  (Tremori,  Baisini,  Enkvist,  Bruzzone,  &Nyce,  2012).  Indeed,  serious  games 
can  be  very  powerful  tools  for  developing  skills  and  analytical  abilities  for  decision-making, 
for  developing  knowledge  and  also  to  change  individual  and  personal  attitudes. 

Serious  games  provide  the  opportunity  for  immersive  practice  so  trainees  can  develop  the 
skills  necessary  for  dealing  effectively  and  efficiently  with  whole-task  decision-making, 
whereas  experiencing  the  overload  and  anxiety  associated  with  real-life  crises  (Sniezek  et 
al.,  2002).  Such  a  virtual  experience  avoids  the  risk  of  costly  error  inherent  in  trial  and 
error  learning,  while  providing  some  of  the  benefits  of  actual  experience,  such  as  facing 


31  March  2014 


30 


Version  1.1 


12om  Process 


ci 


£ 


THALES 


psychological  pressure,  engaging  in  sensemaking,  and  recognizing  when  constraints  such 
as  rules  are  helpful  or  not  (Moynihan,  2009]. 

3.2.3  Develop  and  Integrate  a  New  Facilitator  Role 

The  inclusion  of  an  interagency  planning  specialist/facilitator  (with  modeling  expertise]  in 
a  WoG  team  could  go  a  long  way  for  reducing  the  training  burden  on  the  rest  of  the  team. 
The  version  1.0  of  the  12om  process  does  include  a  facilitator,  but  the  tasks  and  the  precise 
goals  associated  with  this  role  are  yet  to  be  defined.  Task  analysis  methods  focusing  on  the 
12om  methodology  could  help  in  defining  the  exact  role  carried  out  by  the  facilitator.  In 
particular,  while  the  core  features  of  the  IMAGE  v3  tool  were  regarded  as  valuable  for  most 
users,  the  more  complex,  using  the  specialized  features  would  require  a  particularly  well- 
designed  training  package  or  be  supported  by  a  specialized  facilitator.  One  additional 
potential  advantage  of  having  an  interagency  planning  specialist  within  the  team  could  be 
to  ensure  that  one  team  member  is  not  associated  to  a  specific  line  of  operation.  Having 
such  a  team  member  with  no  a  priori  agenda  could  also  help  ensure  a  balanced 
consideration  of  factors.  Finally,  since  the  key  element  to  improve  in  the  12om 
methodology  is  Common  Understanding,  one  of  the  key  roles  of  the  specialist/facilitator 
could  be  to  actively  gage  and  support  this  particular  element  -  particularly  using 
communication  reflection  techniques  such  as  mirroring  and  paraphrasing. 

3.3  Concluding  Remark 

The  12om  methodology’s  toolbox  approach  emphasizes  flexibility  and  provides  a  coherent 
framework  for  complex  problem  solving.  While  some  components  are  very  intuitive  and 
can  be  learned  without  any  major  effort,  it  seems  somewhat  inevitable  that  using  other 
components  will  require  significant  mental  effort  to  succeed  in  understanding  complex 
situations.  Nonetheless,  the  12om  methodology  is  believed  to  succeed  in  facilitating  this 
difficult  process.  Fritz-Millett  (2013]  states  that  practical  tools  and  metrics  are  required  to 
optimise  the  implementation  of  a  comprehensive  approach.  For  instance,  based  on  results 
from  the  Canadian  Army  Experiment  10,  he  notes  that  various  diagramming,  modelling  and 
discourse  capture  toolsets  need  to  be  incorporated  into  the  WoG  approach  to  complex 
problem  solving  and  that  such  toolsets  would  need  to  be  capable  of  adaptation  to  diverse 
group  needs.  The  12om  methodology  provides  just  that,  and  will  hopefully  be  exploited  by 
Canada  and  its  partners  to  positively  impact  future  comprehensive  approach  initiatives. 
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Appendix  A-  Key  Findings  from  LOE#2 

Using  the  MYRIAD  multi-criteria  analysis  tool,  a  preference  model  that  combines  different 
key  results  into  a  coherent  assessment  of  the  overall  12om  methodology  and  its 
components  was  designed.  Furthermore,  MYRIAD  allowed  performing  a  sensitivity  analysis 
that  derives  the  most  promising  areas  for  improvement. 

General  Findings 

Overall,  the  12om  methodology  was  rated  very  favourably,  and  no  components  were 
identified  as  missing  or  as  superfluous.  All  components  were  rated  relatively  favourably  in 
terms  of  the  compromise  between  support  and  feasibility  of  implementation.  The  main 
strength  of  the  12om  methodology  is  in  supporting  the  three  dimensions  of  Collaborative 
Understanding  (i.e.,  collaboration  support,  integration  of  different  perspectives,  common 
understanding)  in  a  very  balanced  way.  Furthermore,  its  support  was  found  to  cover  all 
tasks  associated  with  the  OPP.  The  main  challenge  identified  by  this  analysis  was  the 
overall  need  to  address  the  issue  of  training  time/effort. 

Team  dynamics  analyses  all  pointed  toward  excellent  team  collaboration.  Social  network 
analysis  of  communications  clearly  showed  the  active  leading  role  of  the  J5  (which  was 
expected).  All  participants  save  J5  had  similar  sociometric  status,  emission  degrees  and 
reception  degrees,  revealing  similar  levels  of  participation  during  the  exercise.  Content 
analysis  suggested  that  members  of  the  WoG  team  made  efforts  for  integrating  different 
perspectives  during  the  process  since  the  development-related  content  (which  is  often  left 
apart  in  "integrated”  operations)  was  by  far  the  most  important  topic  at  some  point  during 
the  LOE  #2.  Finally,  the  transactive  memory  system  measurement  instrument  showed  that 
the  members  of  the  WoG  team  perceived  the  other  team  members  as  being  credible  and 
found  that  coordination  within  the  team  was  good. 

Accordingly,  no  major  changes  are  warranted  in  this  third  iteration  of  the  12om  process 
development.  Nonetheless,  analyses  revealed  that  future  work  on  the  12om  methodology 
should  prioritize  changes  that  would  improve  common  understanding.  Note  that  this  is  not 
because  common  understanding  was  poorly  supported  but  rather  because  the  sensitivity 
analysis  showed  that  further  supporting  this  dimension  in  particular  would  have  the  most 
impact  overall. 

Component-Specific  Findings 

Individual  component  assessments  varied  between  63%  and  82%  showing  that  there  is 
still  room  for  improvement,  and  giving  a  magnitude  of  priority  in  terms  of  what 
components  need  the  most  improvement.  Results  for  the  eight  main  components  and  four 
IMAGE  v3  subcomponents  are  summarized  in  Table  Al. 
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Table  Al.  MYRIAD  output  for  the  12om  methodology  components  analysis 


Component 

Score 

OP  Design  Tool 

82% 

Conceptual  Diagrams  (Collaborative) 

76% 

T earn  Building  and  Handover  Procedure 

76% 

Common  Glossary 

76% 

OPP  Handbook 

71% 

WoG  MA  Brief  Template 

70% 

Conceptual  Diagrams  (Individual) 

68% 

CIM 

63% 

Sub-component 

Score 

Creating  a  Common  Vocabulary 

81% 

Sharing  Conceptual  Diagrams 

69% 

Creating  Views  Using  Filters 

68% 

Querying  Using  Filters 

68% 

Note  that  prior  to  LOE  #2  some  component  names  have  been  revised  and  some 
components  split  in  two.  OP  Design  was  divided  into  OP  Design  Tool  and  OP  Design  Process 
(and  since  the  latter  is  part  of  military  doctrine  it  was  not  considered  during  the  12om 
MYRIAD  assessment j.  IMAGE  v3  has  been  divided  into  Conceptual  Diagrams  (Individual] 
and  Conceptual  Diagrams  (Collaborative). 

Only  a  few  sub-components  were  rated  as  being  characterized  by  lower  or  equivalent 
usability  than  similar  tools  in  this  context:  i.e.,  sharing  conceptual  diagrams,  querying 
conceptual  diagrams  using  filters,  and  creating  views  using  filters.  Similarly,  in  terms  of 
cost-benefit  ratings,  the  lowest  ratings  were  associated  with  the  IMAGE  v3  tool  sub¬ 
components:  sharing  conceptual  diagrams  and  querying  conceptual  diagrams  using  filters. 
Taken  together,  the  above  results  suggest  that  the  core  features  of  the  IMAGE  v3  tool  are 
valuable  for  most  users,  but  that  the  more  complex,  specialized  features  should  be  trained 
further  or  supported  by  a  specialized  facilitator.  The  creation  of  a  common  vocabulary  sub¬ 
component  of  IMAGE  v3,  however,  proved  to  be  a  particularly  valued  sub-component  of 
IMAGE  v3. 

A  short  summary  of  the  multicriteria  assessment  results  is  shown  below: 

WoG  OPP  handbook 

The  main  weakness  of  this  component  is  its  relatively  low  impact  on  taskwork,  since  it  is 
mainly  categorised  as  reference  material.  Results  in  terms  of  feasibility  are  relatively  good. 
According  to  the  sensitivity  analysis,  improving  any  of  the  four  dimensions  of  support  is 
deemed  equally  beneficial. 

Team  building  and  handover  procedure 

The  main  weakness  of  this  component  is  its  relatively  low  impact  on  taskwork,  since  it  is 
mainly  categorised  as  reference  material.  Otherwise,  with  the  exception  of  a  moderately 
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demanding  use  time/effort,  results  are  extremely  favorable.  According  to  the  sensitivity 
analysis,  the  priority  areas  for  improvement  are  OPP  support.  Integration  of  perspectives, 
Common  understanding,  and  Impact  on  taskwork. 

Common  glossary 

The  sole  weakness  of  this  component  is  its  relatively  low  impact  on  taskwork,  since  it  is 
mainly  categorised  as  reference  material.  According  to  the  sensitivity  analysis,  the  most 
valuable  areas  for  improvement  are  Impact  on  taskwork,  OPP  support,  Common 
understanding,  and  Collaboration  support 

Conceptual  diagrams  (individual) 

The  main  weakness  of  this  component  is  related  to  feasibility  in  terms  of  time  and  effort 
required  for  training  and  use.  However,  despite  these  difficulties,  this  component  is 
deemed  highly  valuable  especially  because  of  its  important  impact  on  the  taskwork  to 
perform.  According  to  the  sensitivity  analysis,  the  key  areas  for  improvement  are  1)  Any 
one  of  dimension  of  support  (other  than  impact  on  taskwork  which  cannot  be  further 
improved);  2)  Reduce  use  time/effort  (e.g.,  place  a  time  limit  on  list  of  concepts/relations); 
and  3)  Reduce  training  time/effort. 

Conceptual  diagrams  (collaborative) 

The  main  weakness  of  this  component  relates  to  feasibility,  specifically  its  demanding  use 
time/effort.  Nonetheless,  this  component  is  deemed  highly  valuable  because  of  its  high 
positive  impact  on  the  taskwork,  and  is  the  component  with  the  highest  (94%)  satisfaction 
in  terms  of  overall  support  provided.  According  to  the  sensitivity  analysis,  the  key  areas  for 
improvement  are:  1)  Reduce  use  time/effort;  and  2)  Reduce  training  time/effort. 

Cross-impact  method 

The  main  weakness  of  this  component  is  feasibility,  specifically  in  terms  of  its  use 
time/effort.  Nonetheless,  this  component  is  deemed  highly  valuable  in  terms  of  its  impact 
on  the  taskwork  and  received  quite  high  usability  ratings  when  considering  the  nature  of 
the  analytical  method.  According  to  the  sensitivity  analysis  several  possible  areas  for 
improving  the  cross-impact  method  are  equally  viable:  1)  Reduce  use  time/effort;  2) 
Increase  OPP  Support;  3)  Improve  Integration  of  Perspectives;  4)  Increase  support  to 
Development  of  Common  Understanding;  5)  Improve  support  to  Collaboration;  and  6) 
Reduce  training  time/effort. 

OP  design  tool  (DP  analysis) 

With  an  82%  satisfaction  based  on  the  multicriteria  analysis,  this  is  the  most  successful 
component  in  the  12om  methodology.  According  to  the  sensitivity  analysis,  the  four  main 
areas  that  would  benefit  the  most  from  improvement  are:  1)  Impact  on  taskwork  (which 
was  already  relatively  high);  2)  Integration  of  Perspectives;  3)  Common  Understanding; 
and  4)  Collaboration  Support. 

WoGMA  brief  template 

Its  sole  weakness  is  its  relatively  low  impact  on  taskwork,  since  it  is  mainly  categorised  as 
reference  material.  This  component  has  the  greatest  feasibility  (85%)  amongst  the  different 
components  considered  here.  According  to  the  sensitivity  analysis  four  equally  good  areas 
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for  improvement  are:  1]  Impact  on  taskwork;  2)  OPP  support;  3]  Integration  of 
perspectives;  4)  Common  understanding;  and  5)  Collaboration  support. 

Flexibility  and  Impact  of  the  12om  Components 

A  functional  gap  analysis  provided  useful  information  on  the  flexibility  and  impact  on 
taskwork  of  the  12om  methodology  components.  Table  A2  shows  where  the  main  12om 
components  provide  support  and  what  is  the  relative  impact  on  taskwork  it  provides.  Each 
line  represents  a  component,  which  is  plotted  against  CFOPP  sub-tasks  and  12om  objective. 
A  "1"  was  entered  whenever  the  analysis  revealed  that  a  component  was  supporting  a  sub¬ 
task  or  an  objective.  Between  parentheses,  the  number  indicates  the  relative  impact  of  that 
component 


Table  A2.  Summary  of  support  to  OPP  and  12om  objectives  by  methodology  component 

Initiation  Orientation  COA  development  Objectives 


£= 

CD 


WoG  OPP  handbook 

i(i) 

i(i) 

i(i) 

i(i) 

i(i) 

i(i) 

i(i) 

i(i) 

i(i) 

i(i) 

hi) 

i(i) 

hi) 

i(i) 

hi) 

i(i) 

hi) 

17 

17 

WoG  MA  brief  template 

i(i) 

i(i) 

2 

2 

OP  Design  tool 

1(3) 

1(3) 

1(3) 

1(3) 

1(3) 

1(3) 

1(3) 

7 

21 

Cross-impact  method 

1(4) 

1(4) 

1(4) 

1(4) 

4 

16 

Conceptual  diagrams 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

1(4) 

14 

56 

Common  glossary 

mi 

KD 

mi 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

KD 

17 

17 

Team  building  and  handover 

i(i) 

1(1) 

1(1) 

1(1) 

4 

4 

Sum 

2 

3 

2 

4 

5 

4 

4 

4 

6 

4 

4 

4 

3 

4 

4 

4 

4 

65 

Weighed  sum 

2 

3 

2 

10 

13 

10 

6 

7 

14 

9 

9 

9 

6 

9 

7 

7 

7 

133 

The  analysis  shows  that  all  components  do  not  provide  support  throughout  the  whole 
CFOPP.  Some  components,  such  as  the  WoG  OPP  handbook  and  Common  glossary,  are  very 
flexible  as  they  provide  support  to  most  if  not  all  sub-tasks  of  the  CFOPP.  However,  on  the 
other  hand,  they  have  relatively  low  impact  on  taskwork  because  they  are  not  used  directly 
to  generate  a  product.  Moreover,  it  highlights  the  differential  impact  of  the  components  on 
taskwork  to  perform.  For  instance,  it  shows  that  although  OP  Design  tool  is  not  very 
flexible,  it  has  a  strong  impact  on  task  work. 

Table  A2  is  intended  to  provide  information  for  the  selection  of  the  appropriate 
components  to  use  in  a  given  situation.  It  may  help  deciding  on  the  exact  toolset  that  a 
commander  and  interagency  planning  team  members  may  want  to  have  while  carrying  our 
mission. 


31  March  2014 


39 


Version  1.1 


12om  Process 


THALES 


31  March  2014 


40 


Version  1.1 


